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METHOD AND SYSTEM FOR ESTIMATING THE POSITION OF 
MOVING OBJECTS IN IMAGES 

BACKGROUND 

Field of the Invention 

[0001] The present invention relates to the tracking of objects in image data. 
More particularly, the present invention relates to a method and system for 
estimating the position of moving objects in a set of image data. 
Background Information 

[0002] Target Tracking is an essential requirement for surveillance systems 
employing one or more sensors, together with computer subsystems, to interpret 
an environment. The objective of target tracking is to collect sensor data from a 
field of view (FOV) containing one or more potential targets of interest and to 
then partition the sensor data into sets of observations, or tracks, that are produced 
by the same sources. Once tracks are formed and confirmed (so that background 
and other false targets are reduced), the number of targets can be estimated and 
quantities, such as target velocity, future predicted position, and target 
classification characteristics, can be computed for each track. 
[0003] A target tracker can lose track of a target for many different reasons. 
For example, missiles plumes, blurring of the image, the target going behind 
objects such as trees or buildings, and aspect changes are just a few of the possible 
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reasons which can cause a target tracker to lose track of the target. Whatever the 
reason, however, the results are the same. In most circumstances when the target 
tracker indicates that there is insufficient information to track the target, a loss of 
track occurs, which forces that operator to reacquire the target. Thus, operators 
must spend time to reacquire the target track instead of being allowed to 
accomplish other tasks. 

[0004] It would be desirable to provide a simple method that can be used by any 
target tracker for automatically tracking objects that may become occluded for a 
period of time by any type of occlusion. 

SUMMARY OF THE INVENTION 
[0005] A method and system are disclosed for estimating a position of moving 
objects in a set of image data. In accordance with exemplary embodiments of the 
present invention, a position of an object is identified in a first frame of image 
data acquired at a first time. The object is determined to be undetected in a 
second frame of image data acquired at a second time. Movement of the object is 
estimated to determine its estimated position in the second frame of image data 
using at least one of velocity and acceleration of the object and time between 
frames of image data. The estimated position is used to determine a position of 
the object in a third frame of image data acquired at a third time. 
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BRIEF DESCRIPTION OF THE DRAWING FIGURES 
[0006] Other objects and advantages of the present invention will become 
apparent to those skilled in the art upon reading the following detailed description 
of preferred embodiments, in conjunction with the accompanying drawings, 
wherein like reference numerals have been used to designate like elements, and 
wherein: 

[0007] FIGS. 1A, IB and 1C are flowcharts illustrating the steps carried out for 
estimating a position of moving objects in a set of image data in accordance with 
an exemplary embodiment of the present invention; and 

[0008] FIG. 2 illustrates a system for estimating a position of moving objects in 
a set of image data in accordance with an exemplary embodiment of the present 
invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0009] FIGS. 1A, IB and 1C are flowcharts illustrating the steps carried out for 
estimating a position of moving objects in a set of image data in accordance with 
an exemplary embodiment of the present invention. In step 102 of FIG. 1A, a 
position of an object is identified in a first frame of image data acquired at a first 
time. The frames of image data can be any number of pre-stored frames of image 
data which can be selected and retrieved from a database of images or from any 
other electronic storage media. Alternatively, the frames of image data can be any 
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number of frames of image data acquired in any manner known in the art, from 
any of a variety of electro-optical or imaging systems or sensors including, but not 
limited to, a thermal sensor, imaging radar sensor, infrared sensor, Charge- 
Coupled Device (CCD) cameras, Forward-Looking Infrared (FLIR), vidicon 
cameras, Low Light Level cameras, laser illuminated cameras or the like. 
However, any system which collects image data can be used to acquire the frames 
of image data. 

[0010] To keep a record of positional data of the object, in step 104 a database 
is maintained of positional values of the object. According to exemplary 
embodiments, the database can be any type of computer database which stores 
information in any type of electronic storage media for later retrieval. For 
example, the database can reside either internally or externally to any tracker 
system with which the present invention can interface. The positional values of 
the object can include the positional data from the initial position to the last known 
position of the object and can include, for example, the velocity or acceleration of 
the object, or any combination thereof or any additional types of positional 
information. The positional values of the object can be maintained for every 
frame of image data. 

[0011] In step 106, a database is maintained of stabilization values of the object. 
The stabilization values of the object can be maintained for every frame of image 
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data. The stabilized values can be generated by, for example, an Inertial 
Measurement Unit (IMU) or any other type of inertial unit. Those of ordinary 
skill in the art will recognize that the positional values and the stabilization values 
can be maintained for any number of frames of image data. In addition, those of 
ordinary skill in the art will recognize that any or all of the maintained positional 
and stabilization values and any or all of the collected frames of image data can be 
used by exemplary embodiments of the present invention to estimate the position 
of moving objects in a set of image data. 

[0012] Determining the direction of a target while it is occluded is difficult, 
especially for non-IMU data, because a tracker system does not lock onto the same 
exact centroid position of the target in each frame of image data. The tracker may 
lock onto a position that is a few pixels away from the original centroid position. 
Furthermore, while the calculations performed according to exemplary 
embodiments of the present invention can use both integer and fractional-integer 
pixel values to determine the position location of an object in a frame of image 
data, conventional tracker systems can only handle integer pixel values. 
Consequently, the positional values generated by exemplary embodiments of the 
present invention may have to be rounded to the nearest whole integer before 
being passed to the tracker system. This may result in position errors in the 
algorithms used by conventional tracker systems. The presence of 
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rounding/position errors in conjunction with the tracker system not precisely 
locking onto the same position can cause the tracker to have a wobble or other 
types of distortion in it. The wobble and other similar types of distortion can be 
further exacerbated if the background of the imagery is moving. Thus, without 
compensation, it is possible that these errors and distortion may affect the 
accuracy of the directional estimates generated by exemplary embodiments of the 
present invention. 

[0013] To solve the problems associated with wobble and other similar types of 
distortion, exemplary embodiments of the present invention stabilize the image 
data for processing. As will be described hereinafter, the outputs of the 
stabilization process are pixel values that represent how much the background in 
the imagery is moving. If the background is stationary, for example, then these 
values are zero. Exemplary embodiments of the present invention use these values 
to assist in minimizing the effects that the wobble and other similar types of 
distortion can have in determining the direction of movement of the target while it 
is occluded. 

[0014] In step 108, a determination is made as to whether the object is 
undetected in a second frame of image data acquired at a second time. Since most 
conventional target tracking systems can issue a loss of lock indicator when a 
target cannot be tracked, the loss of lock indication can be used to determine 
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whether or not the exemplary steps of the present invention should be performed 
to track an occluded target. However, any indication that a loss of target has 
occurred can be used. In step 1 10, if the track of the object has not been lost by 
the tracker system (i.e., the object is detected in the second frame of image data), 
the positional and stabilization values of the object are updated in the databases of 
positional and stabilization values in steps 104 and 106, respectively. However, 
in step 110, if it is determined that the track of the object has been lost by the 
tracker system (i.e., the object is undetected in the second frame of image data), in 
step 112 the positional values of the object are retrieved from the database of 
positional values. These values can include, for example, both the past and 
present positions of the object or any other desired positional values. In step 114, 
the stabilization values of the object are retrieved from the database of stabilization 
values. 

[0015] If it is determined that the track of the object has been lost by the tracker 
system in step 110 of FIG. 1A, in step 116 of FIG. IB, the movement of the 
object can be estimated to determine its estimated position in the second frame of 
image data using at least one of velocity and acceleration of the object and time 
between frames of image data. According to exemplary embodiments, in step 
118, difference values are calculated between the first frame of image data and the 
second frame of image data for positional values of the object. The resulting 
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values represent the positional difference between frames. In step 120, difference 
values are calculated between the first frame of image data and the second frame 
of image data for stabilization values of the object. In step 122, the stabilization 
difference values are subtracted from the positional difference values for each 
frame of image data to generate stabilized positional difference values. The 
resulting stabilized positional difference values represent the actual true movement 
of the target regardless of the background. 

[0016] In step 124, the data time interval is determined using a time between 
frames of image data. The data time interval can be a time constant that is equal 
to the number of frames between the initial target lock and the last known target 
lock. In step 126, an absolute displacement of the object is determined by 
summing the stabilized positional difference values over the data time interval. 
[0017] In step 128 of FIG. 1C, a constant acceleration of the object is calculated 
during the time interval using a predetermined acceleration function. According 
to exemplary embodiments, the constant acceleration can be calculated using the 
following equation: 

(r -r ) -v *t 

a = 1 ° ° a: 

0.5*r 2 
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In the above equation (1), "a" is the constant acceleration, "r" is the current 
position of the object in pixel space, "r 0 " is the initial position of the object in 
pixel space, "v 0 " is the initial velocity of the object, and "t" is the data time 
interval. For example, the initial position of the object can be near the center of 
the image and the initial velocity can be zero. However, the variables can be any 
values. 

[0018] In step 130, the current velocity of the object is calculated during the 
data time interval using a predetermined velocity function. According to 
exemplary embodiments, the current velocity can be calculated using the following 
equation: 

v=v o +a*t (2) 

In the above equation (2), "Vi" is the current velocity of the object, "v 0 " is the 
initial velocity of the object, "a" is the constant acceleration calculated using 
equation (1), and "t" is the data time interval. 

[0019] In step 132, the estimated movement of the object is calculated from the 
constant acceleration and current velocity using a predetermined position function. 
The estimated movement determines the distance that the object has traveled in the 
next frame of imagery. According to exemplary embodiments, the estimated 
movement can be calculated using the following equation: 
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e i 2 ° ' 

In the above equation (3), "r e " is the estimated movement of the object, "v" is the 
initial position of the object, "v 0 " is the initial velocity of the object, "v," is the 
current velocity of the object calculated using equation (2), and "t" is the data time 
interval. 

[0020] In step 134, the actual movement of the object is calculated by adding 
the stabilization difference values to the estimated movement of the object. The 
estimated movement can be calculated using equation (3). According to 
exemplary embodiments, the stabilization values from the previous frame of image 
data are added to the actual movement. Incorporating the stabilized values into 
the estimated movement allows for a determination of the direction of movement 
of the object while it is obscured. The direction of movement of the object can be 
determined from the sign of the actual movement. Thus, if the value is positive, 
then the object is moving from left to right in the imagery and so will be located 
to the right of its previous position, where pixels increase from left to right. 
However, if the value is negative, then the object is moving from right to left in 
the imagery and so will be located to the left of its previous position. 
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[0021] In step 136, the estimated position of the object in the second frame of 
image data is calculated by adding the actual movement of the object to the 
position of the object in the first frame of image data. Thus, the new estimated 
position of the object can be calculated by adding the estimated distance moved to 
the last known position of the object. Once this is accomplished, in step 138 the 
new positional value is passed out to the tracker system for use in evaluating the 
track function in the next frame of image data. 

[0022] If the tracker is still unable to lock onto the target, then the process can 
be repeated. For example, according to exemplary embodiments, the estimated 
position can be used to determine the position of the object in a third frame of 
image data acquired at a third time. However, the present invention can be used 
to determine the position of the object in any number of frames of image data 
acquired at any number of additional times. Once the object becomes visible 
again, the tracker system can automatically reestablish the track of the object 
again, since exemplary embodiments of the present invention allow the tracker 
system to track the object while it is occluded. 

[0023] A system for estimating a position of moving objects in a set of image 
data in accordance with an exemplary embodiment of the present invention will be 
described with reference to FIG. 2. According to exemplary embodiments, the 
steps of a computer program as illustrated in FIGS. 1A-1C can be performed 
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using an Occluded Target Tracker (OTT) 200. OTT 200 can be a computer, such 
as, for example, a personal computer or any other computer system. OTT 200 
can include a memory 210. Memory 210 can be any computer memory or any 
other form of electronic storage media that is located either internally or externally 
to OTT 200. Memory 210 can store, for example, the steps of a computer 
program as illustrated in FIGS. 1A-1C. As will be appreciated based on the 
foregoing description, memory 210 can be programmed using conventional 
techniques known to those having ordinary skill in the art of computer 
programming to carry out the steps of a computer program as illustrated in FIGS. 
1A-1C described herein. The actual source code or object code for carrying out 
the steps of a computer program as illustrated in FIGS. 1A-1C can be stored in 
memory 210. 

[0024] OTT 200 can be interfaced to any conventional target tracker system 225 
using an interface 230. Interface 230 can be any form of electrical interface, such 
as, for example, an electrical cable providing an RS-232 connection between OTT 
200 and target tracker 225. However, those skilled in the art will appreciate that 
interface 230 is not limited to such an embodiment. For example, interface 230 
can include mechanisms ranging from a simple wire connection to a more complex 
fiber optic connection, with each capable of using different protocols for 
transferring electrical information. In other words, interface 230 encompasses any 
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means for providing a conduit through which electronic image information and 
any other electronic signals can be passed between two electronic devices. 
Alternatively, OTT 200 can be incorporated into any conventional tracker system 
225 by, for example, incorporating the source code or object code for OTT 200 
into the source code or object code of target tracker system 225. 
[0025] Memory 210 can store steps of a computer program to identify a position 
of an object in a first frame of image data acquired at a first time. The frames of 
image data 220 can be any number of pre-stored frames of image data which can 
be selected and retrieved from a database of images or from any other electronic 
storage media. Alternatively, the frames of image data 220 can be any number of 
frames of image data acquired in any manner known in the art, from any of a 
variety of electro-optical or imaging systems or sensors including, but not limited 
to, a thermal sensor, imaging radar sensor, infrared sensor, Charge-Coupled 
Device (CCD) cameras, Forward-Looking Infrared (FLIR), vidicon cameras, Low 
Light Level cameras, laser illuminated cameras or the like. However, any system 
which collects image data can be used to acquire the frames of image data 220. 
The frames of image data can be accessed by both OTT 200 and target tracker 
225. 

[0026] Memory 210 can store steps of a computer program to maintain a 
database of positional values of the object. According to exemplary embodiments, 
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the database can be any type of computer database which stores information in any 
type of electronic storage media for later retrieval. For example, the database can 
reside either internally or externally to any tracker system with which the present 
invention can interface. The positional values of the object can include the 
positional data from the initial position to the last known position of the object. 
The positional values can include, for example, the velocity or acceleration of the 
object, or any combination thereof or any additional types of positional 
information. The positional values of the object can be maintained for every 
frame of image data. Memory 210 can store steps of a computer program to 
maintain a database of stabilization values. The stabilization values of the object 
can be maintained for every frame of image data 220. The stabilized values can 
be generated by, for example, an IMU or any other type of inertial unit. 
[0027] Those of ordinary skill in the art will recognize that the positional values 
and the stabilization values can be maintained for any number of frames of image 
data. In addition, those of ordinary skill in the art will recognize that any or all of 
the maintained positional and stabilization values and any or all of the collected 
frames of image data can be used by exemplary embodiments of the present 
invention to estimate the position of moving objects in a set of image data. 
[0028] Memory 210 can also store steps of a computer program to determine 
that the object is undetected in a second frame of image data acquired at a second 
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time. Since most conventional target tracking systems can issue a loss of lock 
indicator when a target cannot be tracked, the loss of lock indication can be used 
to determine whether or not exemplary embodiments of the present invention can 
be used to track an occluded target. However, any indication that a loss of target 
has occurred can be used. The loss of lock indicator can be passed to OTT 200 
over, for example, interface 230. Memory 210 can store steps of a computer 
program to retrieve positional values of the object from a database of positional 
values and to retrieve stabilization values of the object from a database of 
stabilization values if a determination is made that the object is undetected. 
[0029] Memory 210 can store steps of a computer program to estimate 
movement of the object to determine its estimated position in the second frame of 
image data using at least one of velocity and acceleration of the object and time 
between frames of image data. Memory 210 can store steps of a computer 
program to calculate the difference between the first frame of image data and the 
second frame of image data for positional values of the object and to calculate the 
difference values between the first frame of image data and the second frame of 
image data for stabilization values of the object. Memory 210 can also store steps 
of a computer program to subtract the stabilization difference values from 
positional difference values for each frame of image data to generate stabilized 
positional difference values. 
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[0030] Memory 210 can store steps of a computer program to determine a data 
time interval using a time between frames of image data. The data time interval 
can be a time constant that is equal to the number of frames between the initial 
target lock to the last known target lock. Memory 210 can also store steps of a 
computer program to determine an absolute displacement of the object by 
summing the stabilized positional difference values over the data time interval. 
[0031] Memory 210 can store steps of a computer program to calculate a 
constant acceleration of the object during the data time interval using a 
predetermined acceleration function. According to exemplary embodiments, the 
constant acceleration can be calculated using equation (1). Memory 210 can also 
store steps of a computer program to calculate a current velocity of the object 
during the data time interval using a predetermined velocity function. According 
to exemplary embodiments, the current velocity can be calculated using equation 
(2). Memory 210 can store steps of a computer program to calculate an estimated 
movement of the object from the constant acceleration and current velocity using a 
predetermined position function. According to exemplary embodiments, the 
estimated movement can be calculated using equation (3). 
[0032] Memory 210 can store steps of a computer program to calculate an 
actual movement of the object by adding stabilization difference values to the 
estimated movement of the object. Memory 210 can also store steps of a 
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computer program to calculate an estimated position of the object in the second 
frame of image data by adding the actual movement of the object to the position of 
the object in the first frame of image data. Memory 210 can store steps of a 
computer program to use the estimated position to determine a position of the 
object in a third frame of image data acquired at a third time, if the tracker is still 
unable to lock onto the target. 

[0033] OTT 200 can also include a processor 205 for accessing memory 210 to 
execute the steps of a computer program as illustrated in FIGS. 1A-1C. Processor 
205 can be any known processor, such as, for example, a microprocessor or 
digital signal processor. The system can also include a display 215 that can be 
used in conjunction with OTT 200 and target tracker 225, for example, to display 
to an operator frames of image data and the estimated position of an object in 
those frames of image data. Display 215 can be a computer monitor or any other 
video display device for displaying graphical and/or textual information to a user. 
[0034] The steps of a computer program as illustrated in FIGS. 1A-1C for 
estimating a position of moving objects in a set of image data can be embodied in 
any computer-readable medium for use by or in connection with an instruction 
execution system, apparatus, or device, such as a computer-based system, 
processor-containing system, or other system that can fetch the instructions from 
the instruction execution system, apparatus, or device and execute the instructions. 
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As used herein, a "computer-readable medium" can be any means that can 
contain, store, communicate, propagate, or transport the program for use by or in 
connection with the instruction execution system, apparatus, or device. The 
computer readable medium can be, for example but not limited to, an electronic, 
magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, 
device, or propagation medium. More specific examples (a non-exhaustive list) of 
the computer-readable medium can include the following: an electrical connection 
having one or more wires, a portable computer diskette, a random access memory 
(RAM), a read-only memory (ROM), an erasable programmable read-only 
memory (EPROM or Flash memory), an optical fiber, and a portable compact disc 
read-only memory (CDROM). 

[0035] It will be appreciated by those of ordinary skill in the art that the present 
invention can be embodied in various specific forms without departing from the 
spirit or essential characteristics thereof. The presently disclosed embodiments 
are considered in all respects to be illustrative and not restrictive. The scope of 
the invention is indicated by the appended claims, rather than the foregoing 
description, and all changes that come within the meaning and range of 
equivalence thereof are intended to be embraced. 
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